Automatic Speech Recognition (ASR) is becoming more common in applications such as gaming, web search and other speech understanding systems. Many systems attempt to understand and adapt to different speakers using speaker personalization. Personalization of speech recognition is directed at adapting speech recognition to a specific user. The personalization may use different speaker adaptation techniques (e.g. compensate feature distortions, adjust speech recognition model parameters, and the like). Adapting for speaker variability can be challenging.